Methods and systems for analyzing parts of an electronic file

ABSTRACT

Devices, computer readable medium, and methods for selecting an object displayed on a screen and providing information about that object.

RELATED APPLICATION

This application is a continuation of U.S. Ser. No. 13/122,748, filedMay 16, 2011, entitled “Methods and Systems for Analyzing Parts of anElectronic File”, which is a 371 of International Application No.PCT/US2009/059975, filed Oct. 8, 2009, entitled “Methods and Systems forAnalyzing Parts of an Electronic File”, which is related to, and claimspriority from, U.S. Provisional Patent Application Ser. No. 61/104,077,entitled “Methods and Systems for Analyzing Parts of an ElectronicFile”, filed on Oct. 9, 2008, the disclosure of which is incorporatedhere by reference.

TECHNICAL FIELD

The present invention generally relates to computing devices, softwareand methods and, more particularly, to mechanisms and techniques foranalyzing at least a part of an electronic file.

BACKGROUND

During the past years, the popularity of digital photography isincreasing as digital cameras become more affordable and more powerful,at least in terms of storing images and resolution. Extended databasesof digital pictures are created both for personal and commercialpurposes. Because of the low cost in generating an image as anelectronic file and the convenience of storing this large number ofpictures in electronic form, most people and businesses are takingadvantage of this digital phenomenon.

However, the large amount of data becomes difficult to manage, organizeor index. For example, most of the users would like to be able to tag animage with various pieces of information for remembering differentlandmarks present in the image or for remembering the names of thepersons present in the image. While methods for tagging pictures areavailable, see for example Lloyd-Jones et al., U.S. Patent ApplicationPublication No. 2002/0055955, Simske, U.S. Patent ApplicationPublication No. 2004/0049734, Anderson, U.S. Patent ApplicationPublication No. 2005/0174430, Bhalotia et al., U.S. Patent ApplicationPublication No. 2007/0043748, and Shneiderman, U.S. Pat. No. 7,010,751,the entire disclosures of which are incorporated here by reference, theexisting techniques only allow the user to add predefined boxes atvarious locations of an image and also allow the user to type a desiredtext inside the predefined boxes. All of these techniques superimposethe inserted predefined boxes over the existing pictures and save theadded data as metadata. Thus, the tagging is achieved by associating anX and Y position on the image with the added data, where the X and Yposition on the image is selected by the user with the help of a mouse.In this respect, FIG. 1 shows an image 10 displayed on a screen (notshown). The image includes two persons 12 and 14 and a landmark 16. Theuser, by using the mouse, moves a cursor 18 from a predefined menu 20over the image 10, selecting the X and Y position 22 for the placementof a predefined box. The predefined box 24 is selected from the menu 20and added to the X and Y position 22 as shown in FIG. 2. Then, the usermay type desired text inside the box 24.

According to this approach, the user needs to perform the tagging on acomputer system that offers the above discussed functionalities.Therefore, the tagging operation is limited to computer systems and isnot supported by a TV set. In addition, the tagging operation mapsmultiple predefined boxes to a single image, which limits the analysisof selected parts of the image.

A user may also want to use the large collection of digital images, forexample, to find a same person in different images. In other words, theuser may, for example, decide that he wants to determine all the imagesthat include his mother. One alternative is to look through all theimages and mark those including his mother. This approach is timeconsuming and thus, undesired. Another alternative is to add tags, asdiscussed above, to each picture and to describe within those tags thepersons and landmarks present in the pictures. However, this metadatahas to be entered prior to searching, which is a challenging task.

In another context, the user may also want to be able to buy productsthat are displayed on images presented on a TV set or a computingdevice. These images may incorporate movies, videos, ads, interviews,any kind of information that is presented on a TV set or a computerscreen. In this respect, U.S. Patent Application Publication US2007/0078774, to Brown, the entire content of which is incorporated hereby reference, discloses a method and apparatus for the identification ofproducts in a media program and making such products available forconsumer purchase.

Brown discloses that dedicated devices Beam Back are interposed betweenthe consumer and the device presenting the media program and also thatthe content of the media program is a priori linked to product metadata.The scenes of the media program include identifiers that are linked toother identifiers corresponding to still images, including objectspresented in the media program. When a consumer is watching the mediaprogram, the consumer may press a button on a device to show hisinterest in objects presented in that particular scene. Based on thecorrespondence between the identifiers of the media program and thestill images, the appropriate still images are shown to the consumertogether with the corresponding metadata.

FIG. 3 shows how an image 10 displayed on a screen has selectablepredefined regions 26 and 28 that correspond to advertised objects, apurse 26 and a shoe 28 in this example. The consumer may use the mouseto move the cursor 18 over the purse 26 to request more info about thepurse. This process is possible because a priori links have beenestablished between the purse 26 and the requested information. In thisregard, it is noted that the consumer receives no information about ahat 30 displayed on the image 10 if no prior links have been establishedbetween the hat 30 and the corresponding information.

However, according to this approach, the consumer needs a special deviceand, additionally, both a priori links and product metadata need to bein place before the image is displayed, which might be expensive for theproviders of the content and inconvenient for the consumer.Additionally, it may be desirable to provide methods and systems forproviding information about displayed items without using menus and/ortext boxes, e.g., which enable a user to select an item from a moviewithout distracting overlays being present.

Accordingly, it would be desirable to provide systems and methods thatavoid the above noted limitations of the existing systems.

SUMMARY

According to one exemplary embodiment, there is a method for tagging apart of an image. The method includes providing information fordisplaying the image on a screen; receiving information related topointing a pointing device to the part of the image; generating a borderaround the part of the image, the border corresponding to a movement ofthe pointing device around the part of the image; generating apredefined enclosure, electronically linked to the border, which isconfigured to accept text input from a user; and storing borderinformation and predefined enclosure information together with the partof the image such that the part of the image may be analyzedindependently of the image.

According to another exemplary embodiment, there is a device for tagginga part of an image. The device includes an output configured to provideinformation for displaying the image on a screen; an input configured toreceive information related to pointing a pointing device to the part ofthe image; a processor connected to the input and the output andconfigured to, generate a border around the part of the image, theborder corresponding to a movement of the pointing device around thepart of the image, and generate a predefined enclosure, electronicallylinked to the border, which is configured to accept text input from auser; and a memory connected to the processor and configured to storeborder information and predefined enclosure information together withthe part of the image such that the part of the image may be analyzedindependently of the image.

According to another exemplary embodiment, there is a computer readablemedium for storing computer executable instructions, wherein theinstructions, when executed by a processor, cause the processor toperform instructions for tagging a part of an image. The instructionsinclude providing information for displaying the image on a screen;receiving information related to pointing a pointing device to the partof the image; generating a border around the part of the image, theborder corresponding to a movement of the pointing device around thepart of the image; generating a predefined enclosure, electronicallylinked to the border, which is configured to accept text input from auser; and storing border information and predefined enclosureinformation together with the part of the image such that the part ofthe image may be analyzed independently of the image.

According to another exemplary embodiment, there is a method foridentifying a part of an image in a plurality of pictures. The methodincludes providing information for displaying the image on a screen;receiving information related to pointing a pointing device to the partof the image; generating a border around the part of the image, theborder corresponding to a movement of the pointing device around thepart of the image; extracting pattern recognition information regardingthe part of the image; and storing the pattern recognition informationtogether with the part of the image such that the part of the image maybe identified searched for in other pictures.

According to still another exemplary embodiment, there is a device foridentifying a part of an image in a plurality of pictures. The deviceincludes an output configured to provide information for displaying theimage on a screen; an input configured to receive information related topointing a pointing device to the part of the image; a processorconnected to the input and the output and configured to, generate aborder around the part of the image, the border corresponding to amovement of the pointing device around the part of the image, andextract pattern recognition information regarding the part of the image;and a memory connected to the processor and configured to store thepattern recognition information together with the part of the image suchthat the part of the image may be identified searched for in otherpictures.

According to another exemplary embodiment, there is a computer readablemedium for storing computer executable instructions, wherein theinstructions, when executed by a processor, cause the processor toperform instructions for identifying a part of an image in a pluralityof pictures. The instructions include providing information fordisplaying the image on a screen; receiving information related topointing a pointing device to the part of the image; generating a borderaround the part of the image, the border corresponding to a movement ofthe pointing device around the part of the image; extracting patternrecognition information regarding the part of the image; and storing thepattern recognition information together with the part of the image suchthat the part of the image may be identified searched for in otherpictures.

According to another exemplary embodiment, there is a method forrequesting information about an object present in a media file that isdisplayed on a screen. The method includes receiving information relatedto pointing a pointing device to the object; generating a border aroundthe circled object, the border corresponding to a movement of thepointing device around the object; collecting border information andlocation information, wherein the border information identifies alocation of the border relative to the object and the locationinformation identifies a location of an image in the media filedisplayed on the screen and including the object; transmitting theborder information and the location information to an external server;and receiving from the external server object information not availablein the media file.

According to still another exemplary embodiment, there is a device forrequesting information about an object present in a media file that isdisplayed on a screen. The device includes an input configured toreceive information related to pointing a pointing device to the object;a processor connected to the input and configured to, generate a borderaround the object, the border corresponding to a movement of thepointing device around the object, and collect border information andlocation information, wherein the border information identifies alocation of the border relative to the object and the locationinformation identifies a location of an image in the media filedisplayed on the screen and including the object; an output connected tothe processor and configured to transmit the border information and thelocation information to an external server; and the input being alsoconfigured to receive from the external server object information notavailable in the media file.

According to still another exemplary embodiment, there is a computerreadable medium for storing computer executable instructions, whereinthe instructions, when executed by a processor, cause the processor toperform instructions for requesting information about an object presentin a media file that is displayed on a screen. The instructions includereceiving information related to pointing a pointing device to theobject; generating a border around the object, the border correspondingto a movement of the pointing device around the object; collectingborder information and location information, wherein the borderinformation identifies a location of the border relative to the objectand the location information identifies a location of an image in themedia file displayed on the screen and including the object;transmitting the border information and the location information to anexternal server; and receiving from the external server objectinformation not available in the media file.

According to another exemplary embodiment, there is a method forproviding information about an object, displayed on a screen as part ofan image of a media file, and inquired about by a user by pointing apointing device to the object and drawing a border around the object.The method includes receiving border information and locationinformation, the border information identifying a location of the borderrelative to the object and the location information identifying alocation of the image in the media file displayed on the screen andincluding the object; searching a database for identifying the objectbased on the border information and the location information; populatingthe database with object information if the object is not found in thedatabase; retrieving the object information from the database; andproviding the object information to the user.

According to another exemplary embodiment, there is a device forproviding information about an object, displayed on a screen as part ofan image of a media file, and inquired about by a user by pointing apointing device to the object and drawing a border around the object.The device includes an input configured to receive border informationand location information, the border information identifying a locationof the border relative to the object and the location informationidentifying a location of the image in the media file displayed on thescreen and including the object; a processor connected to the input andconfigured to, search a database for identifying the object based on theborder information and the location information, populate the databasewith object information if the object is not found in the database, andretrieve the object information from the database; and an outputconnected to the processor and configured to provide the objectinformation to the user.

According to still another exemplary embodiment, there is a computerreadable medium for storing computer executable instructions, whereinthe instructions, when executed by a processor, cause the processor toperform instructions for providing information about an object,displayed on a screen as part of an image of a media file, and inquiredabout by a user by pointing a pointing device to the object and drawinga border around the object. The instructions include receiving borderinformation and location information, the border information identifyinga location of the border relative to the object and the locationinformation identifying a location of the image in the media filedisplayed on the screen and including the object; searching a databasefor identifying the object based on the border information and thelocation information; populating the database with object information ifthe object is not found in the database; retrieving the objectinformation from the database; and providing the object information tothe user.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate one or more embodiments and,together with the description, explain these embodiments. In thedrawings:

FIG. 1 is a schematic diagram of a screen in which a graphical userinterface displays an image and a selectable menu;

FIG. 2 is a schematic diagram of the screen of FIG. 1 in which a tag hasbeen added to the image;

FIG. 3 is a schematic diagram of a screen on which an image hasselectable predefined regions;

FIG. 4 is a schematic diagram of a screen on which parts of an image aretagged according to an exemplary embodiment;

FIG. 5 is a schematic diagram of a user interface according to anexemplary embodiment;

FIG. 6 is a schematic diagram of a pointing device that is used toselect an object according to an exemplary embodiment;

FIG. 7(a) is a schematic diagram illustrating how a user circles anobject with the pointing device according to an exemplary embodiment;

FIG. 7(b) depicts an exemplary signal or data packet for use intransmitting selection information according to an exemplary embodiment;

FIG. 8 illustrates the steps of a method for providing information abouta selected object according to an exemplary embodiment;

FIG. 9 is a schematic diagram illustrating how a user may circle part ofthe object with the pointing device according to an exemplaryembodiment;

FIG. 10 is a flow chart illustrating steps of a method for tagging apart of an image according to an exemplary embodiment;

FIG. 11 is a flow chart illustrating steps of a method for identifying apart of an image in another images according to an exemplary embodiment;

FIG. 12 is a flow chart illustrating steps of a method for requestinginformation about a part of an image according to an exemplaryembodiment;

FIG. 13 is a flow chart illustrating steps of a method for providinginformation about an object, displayed on a screen as part of an imageof a media file according to an exemplary embodiment;

FIG. 14 is a schematic diagram of a pointing device according to anexemplary embodiment; and

FIG. 15 is a schematic diagram of a device capable of implementing themethods of FIGS. 10 to 13.

DETAILED DESCRIPTION

The following description of the exemplary embodiments refers to theaccompanying drawings. The same reference numbers in different drawingsidentify the same or similar elements. The following detaileddescription does not limit the invention. Instead, the scope of theinvention is defined by the appended claims. The following embodimentsare discussed, for simplicity, with regard to the terminology andstructure of a TV set having a graphical user interface. However, theembodiments to be discussed next are not limited to these systems butmay be applied to other existing computing systems, such as a computersystem, personal digital assistant, mobile phone, etc.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with an embodiment is included inat least one embodiment of the present invention. Thus, the appearanceof the phrases “in one embodiment” or “in an embodiment” in variousplaces throughout the specification is not necessarily referring to thesame embodiment. Further, the particular features, structures orcharacteristics may be combined in any suitable manner in one or moreembodiments.

Although this application contains a number of interesting exemplaryembodiments, one embodiment that is of particular interest is describedbelow with respect to FIGS. 7 and 8 wherein a user “selects” aparticular item shown as, e.g., part of a program, for example a purseworn by a character in a movie, by drawing a line around that purseusing a free space pointing device. According to this exemplaryembodiment, a signal is transmitted back from the viewing system, e.g.,a television set, set-top box, computer or the like, to a serviceprovider, which signal provides information that identifies the portionof the content which was within the boundary drawn on the screen by theuser. The service provider operates on the data in this signal to flagor identify the object selected by the user, e.g., the particular purseworn by the character. This may be done manually, e.g., by an operator,or automatically via an image recognition system. The service providercan then obtain information about the flagged or identified object,e.g., from a database, and send that information down to the viewingsystem where it can be displayed to the user. Such information caninclude purchase information, more details about the selected object,etc. If the flagged or identified item has not previously been selectedby another user who watched the same content, and if the database doesnot therefore have further information about the selected item, then theoperator may query other data sources, e.g., manufacturer's databases,to obtain such information which may then be used to populate theservice provider's local database to satisfy subsequent user queriesregarding the same item. In this way, users operate to catalog itemsassociated with content which they are viewing and, therefore, thecontent does not have to be pre-tagged or cataloged by the serviceprovider or content provider. That is, according to one exemplaryembodiment, the decisions regarding which of the many products or itemswhich are shown in, e.g., a movie, for which it is important to tag andbe able to provide additional information at a user's request, may beleft until later to be decided by the user's themselves.

Beginning first, however, with FIG. 4, according to an exemplaryembodiment, an image 10 may be displayed on a display (not shown) of anelectronic device. The image may be part of a video, a still picture,etc. The image shows the two persons 12 and 14 and the landmark 16. Theuser, by using a pointing device, as disclosed for example in Liberty etal., U.S. patent application Ser. No. 10/970,811, and Liberty, U.S. Pat.No. 7,158,118, the entire contents of which are incorporated here byreference, may press a predefined button of the pointing device in orderto initiate the tagging process. If the content being displayed on theviewing system or display is dynamic, e.g., a video program, then thepressing of the button (or other any other action which initiates the“tagging” process as described herein) may optionally also have theeffect of causing the pointing device to send a signal to the viewingsystem to automatically pause the displayed content so that the user hasmore time to draw a boundary around the desired object or objects.

Regardless of whether the displayed content is paused or not, the usermay move the pointing device to “circle” any desired part of the image10, person 12 in this example. As a result of the circling, a border orline 32 is drawn around the selected part 12 of picture 10. It is notedthat the meaning of the term “circle” in the exemplary embodimentsincludes an action of the user of drawing a border around a part of animage (figure, etc.) with the help of a pointing device. The border doesnot have to completely surround the selected part of the image and alsothe border does not have to include only the selected object. How tohandle the situation when the user draws a border around two objects inthe image is discussed later. Further, according to an exemplaryembodiment, the user does not have to draw a border that corresponds toa geometric circle as the system can associate a predefined geometricshape (e.g., ellipse, circle, square, rectangle, etc.) to the action ofdrawing a line around the selected object.

The graphical user interface in the electronic device may be configured,according to an exemplary embodiment, to automatically associate apredefined box 24 with the selected border 32. The shapes of the border32 and predefined box 24 may be predefined in the graphical userinterface or may be selected by the user prior to tagging the image.Text or other electronic data may be added by the user to the inside thepredefined box 24. It is noted that the tagging is performed relative toa part of the image 10 and not the entire picture. However, the taggingmay be performed relative to the entire picture 10.

According to another exemplary embodiment, the user may select one ormore other part(s) 16 of the image 10 and draw another border 34 aroundthe other part 16. The another border 34 may be associated with acorresponding predefined box 24. Thus, a single picture 10 may includeone or more parts that are delimited by corresponding borders, eachborder being associated with a predefined box that may include metadataregarding the selected part.

In an exemplary embodiment presented to clarify the concepts introducedabove, assume that a user has a large collection of digital images. Asthe user develops this digital collection, the user may select people orlandmarks of each image, as shown in FIG. 4, and draw borders around theselected people and landmarks. The user may include a description of thelandmark or people in the corresponding metadata holder 24. Then, theuser may analyze parts of the images of the collection, for example, theuser may search for all the images that include his mother given thathis mother was selected in each picture and a description of her wasintroduced in the corresponding metadata holder.

Therefore, according to these exemplary embodiments, an analysis ofparts of the images may be performed instead of analyzing the entirepicture. In one exemplary embodiment, the selected object may beretrieved from the database based on the description of the objectplaced in the metadata holder.

According to another exemplary embodiment, the above discussed taggingmay be implemented in a TV set with or without an external device forsupporting the tagging operation. The TV set may be configured tointeract with the pointing device and the user needs only to move thepointing device around the part to be selected for selecting that part.The initiation of the tagging operation may be performed either bypressing a button on the pointing device or equipping the TV set or topset with a software interface that is displayed on the screen and offersthe user the possibility to select various buttons B that are displayedfrom a selectable menu 36 as shown in FIG. 5. An example of such aninterface is disclosed, for example, in U.S. patent application Ser. No.11/185,220, “GRAPHICAL CURSOR NAVIGATION METHODS”, by F. Wroblewski, theentire disclosure of which is incorporated here by reference.

According to still another exemplary embodiment, the user may have a 3Dpointing device 38 that communicates wirelessly with the TV set 40 asshown in FIG. 6. The user may point the 3D pointing device 38 to thedesired object 12 on the screen of the TV set 40 and may draw a contouraround that object. The contour becomes the border 32, which has acorresponding metadata holder 24. Thus, according to this embodiment,the user may perform the tagging in the comfort of her chair in front ofthe TV set without the need of a computer, mouse and complicatedsoftware for which the user has to remember various commands. Detailsabout (i) an exemplary free space pointing device that is capable ofinteracting with the TV set or any other computational devices, and (ii)the interface between the pointing device and the TV set or othercomputational devices, are disclosed in U.S. patent application Ser. No.11/480,662 “3D POINTING DEVICES”, by Francz et al., the entire contentsof which are incorporated herein by reference.

The metadata information may be inserted in the box 24 either via acomputer equipped with a keyboard and connected to the TV set or anotherlocation where the digital library of pictures is maintained, ordirectly via the pointing device if the TV set includes an interfacethat displays letters and numbers on a certain part of the screen. Then,the user simply points the pointing device to the desired letters andnumbers and by simply pressing a button when a cursor, corresponding tothe pointing device, is over the desired letter, selects the desiredletter or number. Other ways to insert text and numbers may be used aswould be recognized by those skilled in the art.

According to an exemplary embodiment, the user may insert in the box 24a link to one or more desired web sites that, for example, may providemore information about the selected object. For example, when a personis selected, the link may take a viewer of the image to a social website, for example, Facebook®, for providing more information about theselected person.

According to still another exemplary embodiment, a user may use theparts of the image that were selected as described above for anotherkind of analysis. In this embodiment, after the user has “circled” thedesired object, for example, uncle Joe, the user has the option tosearch one or more of the images of the database for uncle Joe, basednot on the metadata that might exist about uncle Joe but based on asimilarity of the face of uncle Joe in one picture with his face inanother picture. In other words, a pattern matching software may beimplemented to analyze the selected object and identify the images inthe database that include a selected pattern, in this case the face ofuncle Joe. The term object is used in the exemplary embodiments toinclude not only per se objects but any element or person selected bythe user. According to a variant of this exemplary embodiment, theselection of the object by the user does not automatically links ametadata holder to the selected object.

Thus, according to these exemplary embodiments, only parts and not theentire picture are used for the analysis and these parts may be selectedby the user. Also, the analysis is not necessarily based on the metadatathat might accompany the parts or the image but may be based on theactual pixels of the image. Pattern matching or recognition software arecommercially available, for example, FACEIT R software, commerciallyavailable from Visionics Corporation of 5600 Rowland Road, Minnetonka,Minn. 55343 U.S.A. and TrueFace software, commercially available fromeTrue, Inc. of 144 Turnpike Road, Suite 100, Southboro, Mass. 01772U.S.A. Other software may be used for performing the patternrecognition.

According to another exemplary embodiment, the concept of circlingdiscussed above may be implemented in a system that allows a user toreceive more information about the circled or selected item, e.g.,including, but not limited to, information associated with purchasingthe selected item. More specifically, as shown in FIG. 7(a), a user mayselect with the pointing device 38 a desired object 26 or 28 that isshown on an image 10 on the TV set 40. The TV set may be replaced withany electronic device that is capable of displaying any image and theimage may be part of a video, TV show, or advertisement that is run onthe screen 40.

It is noted that the desired objects 26 or 28, in this purelyillustrative example a purse and a shoe worn by the actor 12 whileacting in a movie, need not be pre-tagged or pre-linked to any metadataprior to making the movie available for display to end users via acontent distribution system, e.g., VOD, etc., connected to theirtelevisions or other output devices. Also, there is no need according tothis embodiment to provide selection areas on the screen 40, whichcorrespond to the desired objects 26 and 28, such that a user may selectthe objects. In this embodiment, the user simply points the pointingdevice 38 to a desired object 28, draws a border along the desiredobject 28, and the TV set or other computing device generates a signalthat includes an indicator or data which enables a service provider todetermine which part of the screen has been selected by the user. Forexample, the indicator may include at least one of a frame indicator,program indicator, time indicator and information regarding the positionof the boundary relative to the image 10. The actual pixels that werecircled by the user do not have to be known. The television, set-topbox, computer or the like may, for example, transmit a signal or datapacket 700 such as that shown in FIG. 7(b) which includes a frame number702 associated with the video frame in which the object was circled, atime stamp 704, a center point 706 within the bounded area created bythe user's drawing of the boundary line, and optionally one or moreother parameters 708 which define the bounded area (e.g., a radius valuefor a circular area, major/minor axes for an elliptical area,length/width values for a rectangular or square area, etc.). It will beappreciated that the data format illustrated in FIG. 7(b) is purelyillustrative and that, for example, a subset of the illustrated dataelements could be transmitted in signal 700 or additional elements couldbe added.

This information is then transmitted to a call center or serviceprovider, potentially via the same medium that provides the image to theTV set. According to another exemplary embodiment, the information maybe sent to the call center via a different path than the path via whichthe image is received by the TV set. The term “TV set” is used in theseexemplary embodiments for simplicity but one skilled in the art wouldknow that other electronic devices that have a screen may as well beused. The call center may retrieve more information about the objectcircled by the user and provide the user with that additionalinformation.

The steps involved in this exchange of information are discussed in moredetail with regard to the embodiment shown in FIG. 8. According to thisexemplary embodiment a user, while watching a screen of a TV set, maysee, for example, during a movie, an actress wearing an attractive newpurse. If the user wants to receive more information about the purse,e.g., brand name, price, size, where to buy from, the user may point thepointing device, e.g., remote control of the TV set, towards the screenand circle the purse during step 800. According to an exemplaryembodiment, the user may press a specific button of the pointing devicewhile circling the purse, which initiates the “bordering” action. Afterthe user releases the button, information regarding the circled object,e.g., frame information, scene information, time information and theposition of the border relative to the screen, is sent to the callcenter. These actions are illustrated in step 802 under the generic nameof a user requesting information. Other ways for the user to send thisinformation to the call center may be used, as for example, providing aselectable menu on the TV screen in response to pressing a certainbutton on the pointing device. Then, using one or more of the techniquesdescribed above in Wroblewski, the user may activate the buttons of thedisplayed menu to initiate the circling action, without the need topress a button on the pointing device.

The information regarding the location of the purse on the screentogether with the information identifying the scene and/or frame aresent by the TV set (or top box or other device used by the user toreceive the movie) to the call center. At the call center, there may betwo mechanisms for processing the request from the user. According toone exemplary embodiment, wherein no a priori processing of the objectsdesired by the users has been implemented prior to releasing the movieor other content for distribution to the end users, initially there isno information linked to the purse selected by the user.

Thus, a representative of the call center receives the request from theuser in step 803 and, based on the received location information of thescene or the frame in which the purse appeared in the movie, therepresentative can retrieve that scene or frame on his own screen. Therepresentative from the call center has access to all media objects thatare presented on the user's screen. After determining the scene or frameof the movie, based in part on the border information, therepresentative determines the object that the user is interested in, inthis case the purse. Then, the representative decides whether theselected purse has been previously researched or not at this point intime, e.g., by another representative from the call center handlinganother user's request for additional information for that particularitem (potentially from the same frame in the same movie, another framein the same movie, or another piece of content altogether).

Step 803 may also be performed by a computing device. If no user hasrequested information about the purse, then the process advances to step804, in which the request is provided to the same or anotherrepresentative and then to step 806, in which the representativeperforms the research about the purse. The representative may have atable stored on his computing device that associates each object used inthe movie with a corresponding web site of the manufacturer of theobject. By looking up the table for the desired object, therepresentative is presented with the link to the desired web page orother resource. The manufacturer or the operator of the call center maypackage in advance the required information pertinent to the selectedobject so that the representative easily retrieves this information witha click of a button from a database.

Because this is the first time a user has requested information aboutthis object, i.e., the purse, the representative may add in step 810, tothe database, a link between the purse, the scene or frame informationrelated to the purse and the manufacturer information about the purse sothat the next request about the purse may be processed moreexpeditiously. In this way, the database is generated based on theinteraction between the users and the representatives of the callcenter. According to an exemplary embodiment, this process may beimplemented in a computing system to automatically perform the searchand linking. The representative then provides the manufacturerinformation to the user in step 808. This information may be displayedon the screen of the TV or may be provided as an email to the TV oranother device of the user, or may be provided as a voice message to theuser. The representative may also present additional information to theuser, as for example, promotional information regarding related productsof the same manufacturer, ratings of the selected product, places whereto buy the selected product, etc.

However, if in step 803 has been determined that the purse has alreadybeen inquired about by other users, then the representative searches instep 812 the existent database created in step 810 and, as discussedabove, provides the appropriate information to the user in step 814.This information is provided to the user more quickly because therepresentative does not have to collect and assemble the informationabout the purse as this information was already generated in step 810.

One advantage of one or more of the exemplary embodiments discussedabove is that no a priori processing of the movie is necessary, i.e.,the operator of the call center or the owner of the movie does not haveto identify all the selectable objects in the movie based on guess work.According to one or more of the embodiments, the users are selecting anydesired object and together with the representatives of the call centerbuild up a database that include the desired products. Although the userrequesting information about a product that is not in the database mayexperience some delay, because the representative needs to collect livethe information about the required product, the subsequent users do notface this problem. An incentive for the users to be the first to requestinformation about a product may be added by providing a reduction in thepurchase price or other reward, e.g., a monetary reward.

The concept of circling an object on a screen and various challengesassociated with this action are discussed next. If the image displayedon the screen is static or the objects shown in the image move slowly intime, the user has enough time to circle a desired object with the helpof the pointing device to generate the border. However, if the desiredobject moves relatively quickly on the screen, e.g., a character wearingthe purse runs in the movie, it might be difficult for the user tocorrectly circle the purse. Thus, according to an exemplary embodiment,the pointing device or the graphical user interface may be configured topresent optional buttons to the user for slowing down the framespresented on the screen or for actually freezing a desired frame, sothat the user has enough time to perform the circling. An example ofsuch a feature is implemented, for example, by Tivo (2160 Gold Street,Alviso, Calif., 95002).

According to another exemplary embodiment, the user does not have tocompletely circle the desired object but only needs to draw a portion ofthe border, enough to delineate the desired object. In this respect,FIG. 9 shows the portion 42 of the border that is enough for indicatingthe purse. If the portion of the border or the entire border appear tobe ambiguous, i.e., two possible objects are included within the border,the representative or a computing device at the call center informs theuser that two or more objects have been selected, presents the user witha list of the potentially selected objects and the user may select whichspecific object was intended to be selected.

Still according to another exemplary embodiment, the border informationmay include the position of one or more pixels 44, as shown in FIG. 9,that were pointed by the user with the pointing device. If the pointedpixel 44 is inside the purse, the representative or the computing deviceat the call center may still identify in the selected frame the purse.Thus, according to this exemplary embodiment, the user does not need toslow down or freeze the displayed picture for selecting the desiredobject. The user needs only to place, for an instant, the cursorcorresponding to the pointing device on the object and depress a buttonfor tagging the desired object. Using for example the tagging mechanismdiscussed in the previous exemplary embodiments, the TV set of the usermay be configured to add in the metadata holder (predefined box) theinformation related to the frame and/or scene of the selected object andat least the X and Y position of the pixels 44 (if not the entire borderinformation), to be transmitted to the call center.

A set of methods are discussed next with regard to practicing thevarious embodiments discussed above.

According to an exemplary embodiment, the steps of a method for tagginga part of an image. The method includes a step 1000 of providinginformation for displaying the image on a screen, a step 1010 ofreceiving information related to pointing a pointing device to the partof the image and circling at least partially the part of the image, astep 1020 of generating a border around the circled part of the image,the border corresponding to a movement of the pointing device around thepart of the image, a step 1030 of generating a predefined enclosure,electronically linked to the border, which is configured to accept textinput from a user, and a step 1040 of storing border information andpredefined enclosure information together with the part of the imagesuch that the part of the image may be analyzed independently of theimage.

According to another exemplary embodiment, the steps of a method foridentifying a part of an image in a plurality of pictures areillustrated in FIG. 11. The method includes a step 1100 of providinginformation for displaying the image on a screen, a step 1110 ofreceiving information related to pointing a pointing device to the partof the image and circling at least partially the part of the image, astep 1120 of generating a border around the circled part of the image,the border corresponding to a movement of the pointing device around thepart of the image, a step 1130 of extracting pattern recognitioninformation regarding the part of the image, and a step 1140 of storingthe pattern recognition information together with the part of the imagesuch that the part of the image may be identified searched for in otherpictures.

According to another exemplary embodiment, the steps of a method forrequesting information about an object present in a media file that isdisplayed on a screen, are illustrated in FIG. 12. The method includes astep 1200 of receiving information related to pointing a pointing deviceto the object and circling at least partially the object, a step 1210 ofgenerating a border around the circled object, the border correspondingto a movement of the pointing device around the object, a step 1220 ofcollecting border information and location information, wherein theborder information identifies a location of the border relative to theobject and the location information identifies a location of an image inthe media file displayed on the screen and including the object, a step1230 of transmitting the border information and the location informationto an external server, and a step 1240 of receiving from the externalserver object information not available in the media file.

According to another exemplary embodiment, there is a method,illustrated in FIG. 13, for providing information about an object,displayed on a screen as part of an image of a media file, and inquiredabout by a user by pointing a pointing device to the object and drawinga border around the object. The method includes a step 1300 of receivingborder information and location information, the border informationidentifying a location of the border relative to the object and thelocation information identifying a location of the image in the mediafile displayed on the screen and including the object, a step 1310 ofsearching a database for identifying the object based on the borderinformation and the location information, a step 1320 of populating thedatabase with object information if the object is not found in thedatabase, a step 1330 of retrieving the object information from thedatabase, and a step 1340 of providing the object information to theuser.

Thus, according to one or more of the exemplary embodiments, a methodfor analyzing or searching or buying an object that is part of a mediafile and it is displayed on a screen is streamlined and made easier forthe user by providing a pointing device that describes the moves of theuser in a three-dimensional space. The user has the advantage ofpointing the pointing device to the screen of the device that he iswatching, circling (partially or totally) a desired object shown on thescreen and identifying, searching and/or buying that object. Also,another advantage of one or more of the exemplary embodiments is thatthe producers of the media file do not have to a priori load informationabout objects present in the media file that the user may desire. Theprocess is in fact performed a posteriori, after the media file isdisplayed on a screen.

For purposes of illustration and not of limitation, an example of arepresentative three pointing device capable of carrying out operationsin accordance with the exemplary embodiments is illustrated in FIG. 14.It should be recognized, however, that the principles of the presentexemplary embodiments are equally applicable to standard pointingdevices. For example, the pointing device may be a mobile phone, a TVremote control, etc.

The exemplary pointing device 1400 may include a processing/control unit1402, such as a microprocessor, reduced instruction set computer (RISC),or other central processing module. The processing unit 1402 need not bea single device, and may include one or more processors. For example,the processing unit 1402 may include a master processor and associatedslave processors coupled to communicate with the master processor.

The processing unit 1402 may control the basic functions of the mobileterminal as dictated by programs available in the storage/memory 1404.Thus, the processing unit 1402 may execute the functions described inFIGS. 10 to 13. More particularly, the storage/memory 1404 may includean operating system and program modules for carrying out functions andapplications on the pointing device. For example, the program storagemay include one or more of read-only memory (ROM), flash ROM,programmable and/or erasable ROM, random access memory (RAM), subscriberinterface module (SIM), wireless interface module (WIM), smart card, orother removable memory device, etc. The program modules and associatedfeatures may also be transmitted to the pointing device 1400 via datasignals, such as being downloaded electronically via a network, such asthe Internet.

One of the programs that may be stored in the storage/memory 1404 is aspecific program 1406. As previously described, the specific program1406 may interact with a server (call center) to retrieve objectinformation. The object information may include at least one of objectprice information, object specification information, a name of themanufacturer of the object, a name of an outlet where to buy from theobject, a web address of a site that sells the object, and a one clickbutton for buying the object by using the pointing device. The program1406 and associated features may be implemented in software and/orfirmware operable by way of the processor 1402. The programstorage/memory 1404 may also be used to store data 1408, such as theobject information, or other data associated with the present exemplaryembodiments. In one exemplary embodiment, the programs 1406 and data1408 are stored in non-volatile electrically-erasable, programmable ROM(EEPROM), flash ROM, etc. so that the information is not lost upon powerdown of the pointing device 1400.

The processor 1402 may also be coupled to user interface 1410 elementsassociated with the pointing device. The user interface 1410 of thepointing device may include, for example, a display 1412 such as aliquid crystal display, a keypad 1414, speaker 1416, and a microphone1418. These and other user interface components are coupled to theprocessor 1402 as is known in the art. The keypad 1414 may includealpha-numeric keys for performing a variety of functions, includingdialing numbers and executing operations assigned to one or more keys.Alternatively, other user interface mechanisms may be employed, such asvoice commands, switches, touch pad/screen, graphical user interfaceusing a pointing device, trackball, joystick, or any other userinterface mechanism.

The pointing device 1400 may also include a digital signal processor(DSP) 1420. The DSP 1420 may perform a variety of functions, includinganalog-to-digital (ND) conversion, digital-to-analog (D/A) conversion,speech coding/decoding, encryption/decryption, error detection andcorrection, bit stream translation, filtering, etc. The transceiver1422, generally coupled to an antenna 1424, may transmit and receive theradio signals associated with a wireless device.

The pointing device 1400 of FIG. 14 is provided as a representativeexample of a computing environment in which the principles of thepresent exemplary embodiments may be applied. From the descriptionprovided herein, those skilled in the art will appreciate that thepresent invention is equally applicable in a variety of other currentlyknown and future mobile and fixed computing environments. For example,the specific application 1406 and associated features, and data 1408,may be stored in a variety of manners, may be operable on a variety ofprocessing devices, and may be operable in mobile devices havingadditional, fewer, or different supporting circuitry and user interfacemechanisms. It is noted that the principles of the present exemplaryembodiments are equally applicable to non-mobile terminals, i.e.,landline computing systems.

The external server or other systems for providing object information inconnection with the present exemplary embodiments may be any type ofcomputing device capable of processing and communicating objectinformation. An example of a representative server arrangement capableof carrying out operations in accordance with the servers of theexemplary embodiments is illustrated in FIG. 15. The structure shown inFIG. 15 may also be used in a TV set, set top box, etc. Hardware,firmware, software or a combination thereof may be used to perform thevarious steps and operations described herein. The server arrangement1500 of FIG. 5 is an example computing structure that may be used inconnection with such a system.

The server arrangement 1500 suitable for performing the activitiesdescribed in the exemplary embodiments may include a server 1501. Such aserver 1501 may include a central processor (CPU) 1502 coupled to arandom access memory (RAM) 1504 and to a read-only memory (ROM) 1506.The ROM 1506 may also be other types of storage media to store programs,such as programmable ROM (PROM), erasable PROM (EPROM), etc. Theprocessor 1502 may communicate with other internal and externalcomponents through input/output (I/O) circuitry 1508 and bussing 1510,to provide control signals and the like. The processor 1502 carries outa variety of functions as is known in the art, as dictated by softwareand/or firmware instructions.

The server 1501 may also include one or more data storage devices,including hard and floppy disk drives 1512, CD-ROM drives 1514, andother hardware capable of reading and/or storing information such asDVD, etc. In one embodiment, software for carrying out the abovediscussed steps may be stored and distributed on a CD-ROM 1516, diskette1518 or other form of media capable of portably storing information.These storage media may be inserted into, and read by, devices such asthe CD-ROM drive 1514, the disk drive 1512, etc. The server 1501 may becoupled to a display 1520, which may be any type of known display orpresentation screen, such as LCD displays, plasma display, cathode raytubes (CRT), etc. A user input interface 1522 is provided, including oneor more user interface mechanisms such as a mouse, keyboard, microphone,touch pad, touch screen, voice-recognition system, etc.

The server 1501 may be coupled to other computing devices, such as thelandline and/or pointing device, via a network. The server may be partof a larger network configuration as in a global area network (GAN) suchas the Internet 1528, which allows ultimate connection to the variouslandline and/or mobile client devices.

The disclosed exemplary embodiments provide a pointing device, a server,a system, a method and a computer program product for selecting variousobjects displayed on a screen. It should be understood that thisdescription is not intended to limit the invention. On the contrary, theexemplary embodiments are intended to cover alternatives, modificationsand equivalents, which are included in the spirit and scope of theinvention as defined by the appended claims. Further, in the detaileddescription of the exemplary embodiments, numerous specific details areset forth in order to provide a comprehensive understanding of theclaimed invention. However, one skilled in the art would understand thatvarious embodiments may be practiced without such specific details.

As also will be appreciated by one skilled in the art, the exemplaryembodiments may be embodied in a wireless device, a network, as a methodor in a computer program product. Accordingly, the exemplary embodimentsmay take the form of an entirely hardware embodiment or an embodimentcombining hardware and software aspects. Further, the exemplaryembodiments may take the form of a computer program product stored on acomputer-readable storage medium having computer-readable instructionsembodied in the medium. Any suitable computer readable medium may beutilized including hard disks, CD-ROMs, digital versatile disc (DVD),optical storage devices, or magnetic storage devices such a floppy diskor magnetic tape. Other non-limiting examples of computer readable mediainclude flash-type memories or other known memories.

Although the features and elements of the present exemplary embodimentsare described in the embodiments in particular combinations, eachfeature or element can be used alone without the other features andelements of the embodiments or in various combinations with or withoutother features and elements disclosed herein. The methods or flow chartsprovided in the present application may be implemented in a computerprogram, software, or firmware tangibly embodied in a computer-readablestorage medium for execution by a general purpose computer or aprocessor.

1.-93. (canceled)
 94. A device configured to retrieve informationassociated with an image displayed on a screen, comprising: a processorconfigured to generate information indicating a location-of-interest inthe displayed image; and a transceiver configured to: transmit theinformation to an external server; and receive from the external serverobject information identifying an object associated with the indicatedlocation-of-interest based on the information and including attributeinformation indicating attributes of the object.
 95. The device of claim94, wherein the attribute information includes any of: (1) object priceinformation, (2) object specification information, (3) a name of amanufacturer of the object, (4) a name of an outlet to buy the object,(5) a web address of a site that sells the object, or (6) a name of aperson associated with the object.
 96. The device of claim 94, whereinthe transceiver is configured to receive the object information asmetadata that is to be associated with the image.
 97. The device ofclaim 94, wherein the transceiver is configured to receive the objectinformation including any of: (1) additional images associated with theobject; or (2) a link to additional information associated with theobject.
 98. The device of claim 94, wherein the processor is configuredto generate the border information from user input, the generated borderinformation including at least a plurality of pixel coordinates includedin the border.
 99. A method, implemented by a device, of retrievinginformation associated with an image displayed on a screen, the methodcomprising: generating, by the device: (1) border information indicatinga border of an area of the displayed image, the area defining less thanthe entire image; and transmitting, by the device to an external server,the border information to an external server; and receiving, by thedevice from the external server, object information identifying anobject encompassed within the border based on the border information andincluding attribute information indicating attributes of the object.100. The method of claim 99, wherein the attribute information includesany of: (1) object price information, (2) object specificationinformation, (3) a name of a manufacturer of the object, (4) a name ofan outlet to buy the object, (5) a web address of a site that sells theobject, or (6) a name of a person associated with the object.
 101. Themethod of claim 99, wherein the object information is received asmetadata that is to be associated with the image.
 102. The method ofclaim 99, wherein the object information further includes any of: (1)additional images associated with the object; or (2) a link toadditional information associated with the object.
 103. The method ofclaim 99, wherein the generating of the border information includesgenerating the border information from user input, the generated borderinformation including at least a plurality of pixel coordinates includedin the border.
 104. A server configured to send object informationassociated with an image to one or more devices for display, comprising:a receiver configured to receive from a respective device of the one ormore devices: (1) border information indicating a border of an area ofthe displayed image, the area defining less than the entire image, and(2) image identifying information identifying the displayed image; aprocessor configured to determine object information identifying anobject encompassed within the border based on the border information andthe image identifying information; and a transmitter configured to sendto the respective device, the object information including attributeinformation indicating attributes associated with the object.
 105. Theserver of claim 104, wherein the attribute information includes any of:(1) object price information, (2) object specification information, (3)a name of a manufacturer of the object, (4) a name of an outlet to buythe object, (5) a web address of a site that sells the object, or (6) aname of a person associated with the object.
 106. The server of claim104, wherein any of: (1) the object information is sent as metadata thatis to be associated with the image; (2) the object information furtherincludes additional images associated with the object; or (3) the objectinformation further includes a link to additional information associatedwith the object.
 107. The server of claim 104, wherein the processor isconfigured to: determine the attribute information associated with theobject, as first object information; and query one or more databases forthe object information using the determined first object information.108. The server of claim 107, further comprising a storage unitconfigured to store a copy of the image displayed on the respectivedevice, wherein the processor is configured to (1) execute patternrecognition on the area of copied image corresponding to the objectencompassed within the border based on the received border informationto determine the attribute information; or (2) select the attributeinformation from attribute information predetermined to correspond tothe border information for the copied image.
 109. A method for sendingobject information associated with an image to one or more devices fordisplay, the method comprising: receiving, by a server from a respectivedevice of the one or more devices: (1) border information indicating aborder of an area of the displayed image, the area defining less thanthe entire image, and (2) image identifying information identifying thedisplayed image; determining object information identifying an objectencompassed within the border based on the border information and theimage identifying information; and sending, by the server to therespective device, the object information including attributeinformation indicating attributes associated with the object.
 110. Themethod of claim 109, wherein the attribute information includes any of:(1) object price information, (2) object specification information, (3)a name of a manufacturer of the object, (4) a name of an outlet to buythe object, (5) a web address of a site that sells the object, or (6) aname of a person associated with the object.
 111. The method of claim109, wherein any of: (1) the object information is sent as metadata thatis to be associated with the image; (2) the object information furtherincludes additional images associated with the object; or (3) the objectinformation further includes a link to additional information associatedwith the object.
 112. The method of claim 109, wherein the determiningof the object information includes: determining the attributeinformation associated with the object, as first object information; andquerying one or more databases for further object information using thedetermined first object information.
 113. The method of claim 109,further comprising storing a copy of the image displayed on therespective device on the server, wherein the determining of theattribute information includes: (1) executing pattern recognition on thearea of the copied image corresponding to the object encompassed withinthe border based on the received border information to determine theattribute information; or (2) selecting the attribute information fromattribute information predetermined to correspond to the borderinformation for the copied image.